Providing a partial column defect map for a full frame image sensor

ABSTRACT

A method for determining defects in an image sensor, which identifies one or more partial column defects, including capturing a digital image using the image sensor and storing such digital image in a memory; identifying at least one column in the digital image which has corrupted data caused by one or more defective pixels in the image sensor; processing the digital image data from the at least one identified column using an extended differentiation filter; and using the output of the extended differentiation filter to identify a starting position or an ending position of the partial column defect.

FIELD OF THE INVENTION

The present invention relates to full frame image sensors and, moreparticularly, to the identification of partial column defects in suchfull frame image sensors to produce a defect map for concealing suchdefects.

BACKGROUND OF THE INVENTION

A full frame image sensor is basically a two dimensional array of pixelsensing elements of size x columns by y rows. The image sensors capturelight and stores the light captured in the individual pixel sensors. Thepixels are vertically shifted down each column in parallel by one row,with the last row being shifted out and filling a horizontal shiftregister. These pixels in the horizontal shift register are then shiftedout one at a time (serially) until the horizontal shift register iscompletely empty. At this time, the sensor is ready to fill thehorizontal shift register again, and the process of parallel to seriesshift explained above is repeated one row at a time until all rows ofthe sensor have been transported out of the sensor.

In typical high resolution image sensors, some of the pixels of theimage sensing array provide corrupted data, which can be classified intothree different types: pixel, column, and cluster defects. These defectsare often characteristics of the device and are formed during themanufacturing process. The defects are typically mapped during themanufacturing process, but in some cases additional defects are alsodetected when the sensors are assembled into the final product, such asa digital camera. For example, the temperature or the clock and timingcharacteristics of the electronics controlling the sensor can causeadditional defects. Also, during the product assembly, dust, dirt,scratches, etc. may be introduced.

With full frame image sensors, there is often a problem where there areone or more defective pixels in a column. Such a defective pixel willcause corrupted data in the digital image after it is read out of theimage sensor. Depending on the cause of the defect, the column might beonly partially corrupted. For example, the lower portion of the columncould provide proper data while the upper portion provides corrupteddata. To produce the highest quality image, this partial column defectneeds to be identified, so that the data provided by the defective upperportion can be concealed, while the data from the lower portion isutilized to create the final image.

What is needed is a method of quickly and automatically identifyingpartial column defects in an image sensor, so that an effective map ofthe defective pixels can be provided.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to automaticallydetermine the start and end points of a partial column defect in animage produced by an image sensor, such as a full frame image sensor,and to map such corrupted data so that it can be corrected in a digitalcamera.

This object is achieved by a method for determining defects in an imagesensor, which identifies one or more partial column defects, comprisingthe steps of:

a) capturing a digital image using the image sensor and storing suchdigital image in a memory;

b) identifying at least one column in the digital image which hascorrupted data caused by one or more defective pixels in the imagesensor;

c) processing the digital image data from the at least one identifiedcolumn using an extended differentiation filter; and

d) using the output of the extended differentiation filter to identify astarting position or an ending position of the partial column defect.

ADVANTAGES

It is an important advantage of the present invention to provide adefect map of partial column defects, which can be used by a defectcorrection routine that will conceal defects.

It is an additional advantage of the present invention to quickly andaccurately identify partial column defects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram that shows the types of column defectsthat can be found in a digital image produced by an image sensor;

FIG. 2 is a block diagram of a test system for testing an image sensorin accordance with the present invention for automatically identifyingcorrupted data in a digital image and providing a defect map which canbe used in a digital camera to correct such corrupted data;

FIG. 3 is a block diagram of a digital camera, which can be used tocapture the image as shown in the test system of FIG. 2, and also storethe defect map as created by the algorithm of FIG. 4; and

FIG. 4 is a block diagram of the algorithm used in the system of FIG. 2for producing the defect map.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 depicts a schematic diagram to represent the different types ofcolumn defects observable in an image sensor, such as a full frame imagesensor. For a more detailed description of the operation and structureof CCD image sensors, refer to Solid-State Imaging with Charge-CoupledDevices by Albert J. P. Theuwissen.

FIG. 1 depicts an image sensor with both non-defective columns andpartial column defects. A normal column with uncorrupted data 10 isclassified as non-defective. There are basically two types of partialcolumn defects observed in an image. First, the corrupted data may startanywhere (including the top) and end at the bottom of the observeddigital image as shown by columns 12, 14, and 16. Second, the corrupteddata may start at the top and “end” in the middle as shown by defectivecolumn 18.

The first type is common and can have several variations. When viewed inthe observed digital image, the corrupted data can appear to start at apoint and all subsequent pixels can appear corrupted at the samecorrupted light level as shown in column 12. This can be caused by apixel defect and/or channel restriction. This channel restriction ordefective pixel causes the pixel of a column of a full frame sensor, andany subsequent pixel information that passes through such a restrictionor defective pixel, to yield corrupted data in the output digital image.Corrupted data can also appear to fade in and out as shown in column 14.It is believed that a small charge trap in the shift register that skimscharge from one or more pixels and later emits them causes this. Also,if the image sensor uses a color filter array (CFA), such as the BayerCFA pattern shown in commonly-assigned U.S. Pat. No. 3,971,065,corrupted data can appear to be in one color plane only, as shown indefective column 16. This could be due to exposure conditions, andbecause of the channel restriction anomaly explained above, the lightlevel in one of the color planes is being clipped and the resultingoutput image data in that color plane looks corrupted.

The second type of partial column defect, shown in 18, is uncommon andthought to be caused by a break (open circuit) in the lateral overflowdrain (LOD). The LOD is supposed to drain excess electrons, but insteada portion of the LOD potential is floating and injecting charge alongthe column back into the pixels. Although this can be seen in theobserved image as a partial corrupted column, the column could show upas a fully corrupted column, depending on exposure conditions. Incertain exposure conditions, the pixel will not appear to be defectiveand in certain other conditions, the pixel will appear to be defective.Since the root cause of the defect is common to all the pixels in thecolumn, it is better to map this type as a full column rather than as apartial column.

The present invention identifies all of these types of partial columndefects and provides an appropriate defect map for concealing thesedefects.

Turning now to FIG. 2, a representative test system is used to acquirean image, process the image and identify corrupted data, and store thecorrupted data back into the digital camera as a defect map. The testsystem includes an illumination source 20, which directs light through atransparent diffuse target 22 used to produce a flat field image. Thelight intensity is regulated through a filter assembly 24 includingseveral neutral density filters. Filter selection is controlled by ahost computer 42. Parts 20, 22, and 24 are all enclosed in a light boxtest fixture 25 to block unwanted light interference from the outside.The flat field image produced in the light box test fixture 25 iscaptured and processed by a digital camera 30 (described later). Thedigital camera 30 is automatically controlled by the host computer 42.The host computer 42 controls both the capture and retrieval of theimage from the digital camera 30 via an electrical interface, such asone made in accordance with the well-known IEEE 1394 standard.

Once the image has been retrieved from the digital camera 30, a testalgorithm 40 (described later), which has been input to the hostcomputer 42 prior to the beginning of the test, is used to process andanalyze the image. The host computer 42 determines the defect mapaccording to the test algorithm 40 and lists the results on the outputdisplay 44. A defect map is also sent and stored in the digital camera30.

FIG. 3 depicts a block diagram of a digital camera used to capture animage produced by the test system of FIG. 2 described above and storethe defect map produced by test algorithm of FIG.4 (described later). Asstated above, the host computer 42 automatically controls the operationof the digital camera 30. The host computer 42 sends the digital cameraa series of commands via firewire interface in accordance with the IEEE1394 standard. The control interface processor 70 interprets thesecommands and in turn sends commands to a photo Systems interface 80,which sets the exposure control parameters for the digital camera 30.Connected to the photo systems interface 80 are an aperture driver 82and a shutter driver 84. The camera includes an optical lens 50, whichreceives the incoming light. Through the aperture driver 82 and theshutter driver 84, aperture 51 and shutter 52 are controlled,respectively, and allow the incoming light to fall upon the full frameimage sensor 60. The image sensor 60, which may be a KAF-6303E imagesensor manufactured by Eastman Kodak Company, Rochester, N.Y., isclocked by the sensor drivers 62. The output of the image sensor 60 isamplified and processed in a CDS (correlated double sampling circuit) 64and converted to a digital form in an A/D converter 66. The digital datais transferred to processor section 69, which includes a digital imageprocessor 72 that utilizes instructions stored in EEPROM firmware memory74 to process the digital image. Finally, the processed digital image isstored using a memory card interface and removable memory card 78, whichcan be made in accordance with the PCMCIA 2.0 standard interface, or theimage is transferred back to the host computer 42 shown in FIG. 2 viathe firewire interface 76.

The present invention provides an automated test method for effectivelydetecting the endpoints of partial column defects in the image sensor60. FIG. 4 depicts a flowchart of a preferred embodiment of a testalgorithm for operating the system of FIG. 2 to detect partial columndefects. Referring to the flowchart of FIG. 4, the test starts in block90 when the digital camera 30 is connected to the host computer 42, andproperly positioned relative to the light box test fixture 25.

In block 92 an image is captured with the digital camera 30 with a fullframe sensor on a flat field target. Also in block 92 the image istransferred from the digital camera 30 as described above to the hostcomputer 42 where the analysis of the image takes place. As discussedearlier in reference to FIG. 1, there are several different causes andsources of column defects which can cause the defective columns to bebrighter or darker than the surrounding image. For this reason, it ispreferred to have several images taken at different exposure levels,each one to be analyzed separately. Typically a low, mid-range, and highexposure image for each gain setting (e.g each effective ISO setting) ofthe camera is sufficient. The final defect map includes the defectivepartial column defects identified for each of these exposure levels.

In block 94, the host computer 42 performs an algorithm to identify andstore the column addresses of full column defects in the transferredimage. A preferred algorithm for identifying the defective columns is tocompare the mean average of each column to its neighbor. By comparingthe mean average of each column to the mean average of three neighboringnon-defective columns, the relative error is calculated using theformula:δ=(X 0−X)/X  (equation 1)wherein:

-   -   δ is the relative error;    -   X0 is the mean average of column being tested; and    -   X is the mean average of 3 neighboring good columns.

This relative error is then compared to a threshold to determine whetheror not the column has corrupt data in it. If the relative error is abovea limit threshold, which is typically around 10 percent in the preferredembodiment, but will vary depending on gain setting (effective ISOspeed), exposure level, and camera type, the column is marked asdefective and the column address is stored for later analysis. In thismanner, the column addresses of all columns containing partial columndefect columns are identified and stored. At this stage, all the columnshave unknown endpoints. It is the job of the rest of the algorithm todetermine the endpoints.

In block 96, after a defective column list is obtained, the first columnarray of pixels is extracted. Since the image sensor 60 has a Bayer CFApattern and the color pixel values have not been interpolated, therewill be two color planes for each column extracted, which will be eithera green plane and a red plane, or a green plane and a blue plane. Eachcolor plane will be a separate array of data, corresponding to the oddand even rows of the image sensor array.

In block 98, one of the two color planes for the identified column isretrieved from memory by the host computer 42 so that the image data forthat color record in the identified column can be processed.

In block 100, a median filter is performed on the array. This helpsreduce noise and eliminates any pixels that deviate greatly from theothers. In the preferred embodiment, the median filter performs thefollowing operation:Y[i]=Median{X[i−r], X[i−r+1], . . . , X[i], . . . , X[i+r−1],X[i+r]}  (equation 2)wherein:

-   -   Y is output median filtered pixel data array;    -   i is index of array;    -   X is input pixel data array; and    -   r is filter rank;    -   wherein the median is defined as the middle element in a sorted        array.

In the preferred embodiment the filter rank r is 3. Using equation 2above, when r equals 3, then each pixel element i has a median filterperformed on it using an array subset X of size 7.

In block 102, the median filtered column image data is processed usingan extended differentiation filter. An extended differentiation filteris a class of filters that has been found to be particularly suited foridentifying the starting and ending locations of partial column defects.Unlike a discrete derivative function given by the relationship:dp(i)/dy=*dy(p(i+1)−p(i−1))  (equation 3)wherein:

-   -   dp is rate of change of p;    -   i is index of array;    -   dy is rate of change of y; and    -   p is the input pixel data, an extended differentiation filter        uses the mean average of a plurality of pixels, rather than just        a single point, wherein the plurality of pixels provides a        filter of width x in equation 4 below.

The formula for a preferred form of the extended differentiation filteris given by the equation:dp(i)/dy=*dy(μ[p(i+1,i+2, . . . i+x)]−μ[p(i−1, i−2, . . .i−x)])  (equation 4)wherein:

-   -   dp is rate of change of p;    -   i is index of array (corresponding to every other of the sensor        array, since the processing is performed one color plane);    -   dy is rate of change of y;    -   p is the pixel data following median filtering;    -   μ is mean average; and    -   x is constant value equal to the filter width.

An advantage of using an extended differentiation filter is that itsmoothes out the signal, which in many images is quite noisy and whichwould otherwise provide an unreliable indication of the starting andending points of the partial column defects. While the preferred width xfor the KAF-6303E sensor having 2056 rows is 50 pixels, in general, thewidth of the filter depends on the number of pixels in the array, themagnitude of noise in the image, and the signal-to-noise ratio of thearray of data. The minimum width possible is desired because pixels thatdo not have the required width will be set to zero so as to minimizeerrors in the resulting differentiation. The pixels set to zero areenforced at the edges of the column, or in other words the ends of thearray, where the pixels do not have the required filter width on eitherside. This is the safest method for dealing with edge pixels. Performingan erroneous calculation could potentially determine an inaccurate startpoint within the defective column.

It should be noted that from an image quality standpoint, it is worse tomap not enough of the column rather than too much. For example, theactual starting point of the partial column may be in the middle or topof the image. If the extended differentiation filter determines thisstarting point to be near the edge at the bottom, some of the corrupteddata will not be mapped and will be obvious when rendered in a finalimage.

In step 103, the row address of the maximum value of the output of theextended differentiation filter is captured and stored as the tentativestarting point of the partial column defect. The next step is todetermine whether or not this tentative starting point is valid, and ifso, which part of the column is defective.

In order to do this, a comparison with a good neighbor column isperformed in block 106. Two mean averages of a good neighbor column andtwo mean averages of the defective column of a given width of pixels(using an averaging filter similar in size to the extendeddifferentiation filter used in block 102), before and after the startingpoint, are determined.

The preferred averaging filter is given by the equation:μ=Sum{X[i]/n}  (equation 5)wherein:

-   -   μ is mean average;    -   i is index of array;    -   X is the input pixel data array after median filtering; and    -   n (filter width) is number of pixels in X.

Defining the tentative starting point as i0, X is pixel data from i0+1to i0+n for the first mean average and i0−1 to i0−n for the second meanaverage.

Two relative errors are then calculated in block 104 using equation 6,using as inputs the mean average values just calculated using equation5, whereinδ=(X 0 −X)/X  (equation 6)wherein:

-   -   δ is the relative error;    -   X0 is the mean average of the given width of pixels of the        defective column; and    -   X is the mean average of the given width of pixels of the good        column.        Note that the comparison is done on the mean averages of the        columns. Two relative errors are calculated. The first compares        the mean average of a good column before the starting point to        the mean average of a defective column before the starting        point, and the second compares the mean average of a good column        after the starting point to the mean average of a defective        column after the starting point. Relative error measurements        need to be used rather than absolute error due to the        non-uniformity introduced from the test system of FIG. 2.        Non-uniformity is introduced with lens roll-off or light source        impurities, and for practical applications the flat-field image        created is not truly “flat-field”.

In block 106 the relative errors are compared with a threshold todetermine whether the difference between the good column and defectivecolumn is large enough to constitute a defect. This threshold is equalin value to the one used in detecting the column defect of equation 1above.

If the relative errors are not greater than the threshold, then an edgereal flag for this color plane is cleared in block 112. The use of thisedge real flag will be described later in reference to block 120.

If the relative error is greater than a threshold, then an edge realflag is set in block 108 for this color plane and the relative errorbefore the starting point and after the starting point are compared inblock 110.

If the relative error before the starting point is not less than therelative error after the starting point, the column defect starts fromthe top as indicated in block 114. As discussed earlier in thediscussion of FIG. 1, all partial columns starting from the top edgemust be recorded as full columns because any other endpoint would beunreliable.

If the relative error before the starting point is less than therelative error after the starting point, the column defect starts fromthe starting point and ends at the bottom as indicated in block 116. Theendpoints for this color plane of the first identified defective columnhave now been determined and recorded in the memory of the host computer42.

As shown in FIG. 1, it is possible to have a defective column 16 in onlyone color plane. For a Bayer pattern there are two color planes toanalyze for each column of pixels, therefore in block 118 this is notthe last color plane and the process described above for getting a colorplane and analyzing to determine the endpoints is repeated for thesecond color plane of the first identified defective column.

After it has been determined that the last color plane has been analyzedin block 118, the maximum size column may be calculated and recorded tothe defect map in block 120. This is done by comparing the endpointsfrom the color planes that have their edge real flag set. If only one ofthe color planes has its edge real flag set, then the endpoints fromthat color plane are recorded. If both color planes have their edge realflags set, then the maximum end point and minimum start points arerecorded to provide the row addresses of the longest column. If bothcolor planes have their edge real flags cleared, then the column isrecorded as a full column defect.

In block 122, the program then checks to see if this is the lastdefective column, and if not continues on to the next column repeatingthe process starting at block 96. Finally, after the last column hasbeen analyzed in block 122, the test finishes in block 124. At thispoint, a final defect map with all the columns and their respectiveendpoints has been recorded and stored in the memory of the hostcomputer 42 of FIG. 2. The defect map is now transferred from the hostcomputer 42 to the digital camera 30 and stored in the EEPROM firmwarememory 74.

As described relative to FIG. 4, the present invention first identifiesthe column defects (block 94), and then processes the digital image datafrom the identified column using an extended differentiation filter(block 102), and uses the output of the extended differentiation filterto identify starting or ending positions (block 103) of the partialcolumn defect. Partial columns are assumed to have one endpoint at thebottom or top. If the image sensor has a partial column that neitherbegins at the top nor ends at the bottom, then the partial column defectidentified using the algorithm described provides a column defect mapthat provides concealment of a larger column, by extending defect to thebottom of the image sensor array.

A computer program product, such as a readable storage medium, can storethe programs in accordance with the present invention for operating themethods set forth above. The readable storage medium can be a magneticstorage media, such as a magnetic disk (such as a floppy disk) ormagnetic tape; optical storage media, such as an optical disk, anoptical tape, or a machine readable bar code; solid state electronicstorage devices, such as a random access memory (RAM) or a read onlymemory (ROM); or any other physical device or medium employed to storecomputer programs.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   10 normal column-   12 partial column defect-   14 partial column defect-   16 partial column defect-   18 partial column defect-   20 illuminator-   22 target-   24 filter assembly-   25 light box test fixture-   30 digital camera-   40 test algorithm-   42 host computer-   44 output display-   50 optical lens-   51 aperture-   52 shutter-   60 image sensor-   62 sensor drivers-   64 analog gain and CDS-   66 A/D converter-   69 processor section-   70 control interface processor-   72 digital image processor-   74 EEPROM firmware memory-   76 firewire interface-   78 removable memory card and interface-   80 photo systems interface-   82 aperture driver-   84 shutter driver-   90 block-   92 block-   94 block-   96 block-   98 block-   100 block-   102 block-   103 block-   104 block-   106 block-   108 block-   110 block-   112 block-   114 block-   116 block-   118 block-   120 block-   122 block-   124 block

1. A method for determining defects in an image sensor, which identifiesone or more partial column defects, comprising the steps of: a)capturing a digital image using the image sensor and storing suchdigital image in a memory; b) identifying at least one column in thestored digital image which has corrupted data caused by one or moredefective pixels in the image sensor; c) processing the digital imagedata from the at least one identified column using an extendeddifferentiation filter; and d) using the output of the extendeddifferentiation filter to identify a starting position or an endingposition of the partial column defect.
 2. The method of claim 1 furtherincluding the step of: e) providing a defect map which identifies theposition of the defective column and the starting or ending position ofthe partial column defect, so that such defect map can be used in adigital camera to correct the partial column defect.
 3. The method ofclaim 1 wherein the output of the extended differentiation filter iscompared to a threshold determined from the digital image data of atleast one neighboring column which does not include defects.
 4. Themethod of claim 3 wherein step c) includes median filtering the digitalimage data from the at least one identified column prior to processingthe digital image data using the extended differentiation filter.
 5. Themethod of claim 4 wherein the image sensor is a color image sensor, andstep c) is performed on the separate color plane image data of the atleast one identified column.
 6. A computer program product comprising acomputer readable storage medium having a computer program storedthereon for implementing the method of claim 1.